Bhyve-ready NetBSD Images

Images are available for NetBSD 9.3 and contain the following:

• Base OS install, plus compiler and headers, plus man pages on a 10GB image.
• Virtio disk and networking
• Integration using cloud-init configured for SmartOS/Triton (root password and ssh key, data disk preparation, etc)
• mdata client installed, example use is in /etc/cloud/cloud.cfg
• pkgsrc/pkgin installed and ready to go
• Serial console configured for use with vmadm console
• UEFI boot

root@headnode# imgadm source -a http://images.welliver.org

Here's the current list of images:

root@headnode# imgadm avail | grep netbsd
3a362e0e-afaf-11ed-97b3-cfb1bff0c1d0  netbsd-9.3-cloudimage           20230219                                    bsd      zvol          2023-02-19

These images are intended to be light weight, so only the minimum install sets (plus the compiler and headers, necessary for building software). Here's an example of how to add other installation sets:

root# cd / && curl -L http://mirror.planetunix.net/pub/NetBSD/NetBSD-9.3/amd64/binary/sets/ttxt.tar.xz | tar xzvf -

I've tested both images as a pkgsrc build platform and they work at least as well as my previous virtual environment running on VMWare.

An important note regarding security

By default, NetBSD does not assign a password to the root user. This means that unless the password is changed, a) anyone with console access can log in as root and b) anyone in the wheel group can su to root without being prompted for a password.

The images generated will automatically set a root password if it's provided in the instance metadata. This happens automatically in Triton, but it also works with SmartOS if it's provided in the instance information (see the example below for how to do this).

Also note, the metadata client is executable by anyone, so even if the password is set, it's easy to retrieve, unless it's been changed. This seems to be the case with all image types, and the only protection comes from the permission set on the metadata socket or device.

See the metadata client source for more information about what those paths are.

How the images are built

The image build script must be run from a netbsd system, in order to format the ffs disk and install the netbsd boot loader. So, start by cloning the following repository:

https://git.sr.ht/~hww3/netbsd-cloud-images

Edit the script for the version of NetBSD you'd like to use, and then run build.sh. This should create a new blank UEFI-based NetBSD install with everything configured for the first run. The result will be a raw image called "final.raw".

Copy this to your smartos/triton headnode and create a blank bhyve vm:

# create a blank vm setup to start bhyve with EFI boot.
# also, note the example of how to pass ssh keys and root password
# to cloud-init if you're not running triton.
UUID=$(uuid)
export UUID
vmadmin create << EOF
{
  "brand": "bhyve",
  "alias": "install-bhyve",
  "uuid": "${UUID}",
  "vcpus": 1,
  "autoboot": false,
  "bootrom": "uefi",
  "ram": 4096,
  "disks": [
    {
      "boot": true,
      "model": "virtio",
      "size": 10240
    }
  ],
  "nics": [
    {
      "nic_tag": "hacienda",
      "ip": "dhcp",
      "primary": "true",
      "model": "virtio"
    }
  ],
  "internal_metadata": { "root_pw": "f00Bar" },
  "customer_metadata": {
    "root_authorized_keys": "ssh-rsa YOURPUBLICKEY foo@bar.org"
  }
}
EOF

# write the raw image to the zvol of our blank vm.
dd if=/path/to/final.raw of=/dev/zvol/dsk/zones/${UUID}/disk0

# if we feel confident that all will be well, we can take the zfs dump right now
# otherwise, we'll need to test first, then destroy the vm and recreate it  in order
# be able to get a zfs dump that doesn't have a bunch of data from having run the 
# initial bootup. It's not enough to re-dd the disk without first destroying the vm, as the disk will
# contain a bunch of data beyond the end of our raw disk image.
zfs set quota=100G zones/${UUID}
zfs snap zones/${UUID}/disk0@imgsnap
zfs send zones/${UUID}@imgsnap | gzip > netbsd-image-ver.zfs.gz 
vmadm start

# test vm, make sure everything works as expected including cloud-init.
# escape sequence is ~. 
vmadm console ${UUID}

# triton seems to dislike having a rogue instance running on a machine it controls, so 
# while it exists, provisioning of new vms will fail.
vmadm stop ${UUID}
vmadm destroy ${UUID}

Adapted from the following source: https://github.com/virt-lightning/netbsd-cloud-images

Notes for creating the initial image:

NetBSD DVD install images are UEFI ready starting with NetBSD 9.0, so we can finally use BHYVE for the entire process. A key trick is to make sure that the CD drive is located at a high PCI slot to ensure it gets priority:

UUID=youInstanceUUID
ISO="NetBSD-9.3-amd64.iso"
cp $ISO /zones/$UUID/root/
vmadm update $UUID bhyve_extra_opts="-s 7:0,ahci-cd,/$ISO"
vmadm start $UUID
vmadm update $UUID bhyve_extra_opts=""

You can use VNC to perform the original install. Not all VNC clients work equally well. I've had good luck with TigerVNC, but not with MacOS Screen Sharing and the "official" VNC client. Chicken also worked well, but is no longer supported on recent MacOS.

Not categorized | RSS Feed | BackLinks